Papers with expert pruning
SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget (2024.acl-long)
Copied to clipboard
| Challenge: | Mixture of experts (MoE) is a popular technique to improve capacity of Large Language Models (LLMs) but memory-constrained devices are a major concern in edge AI training and serving. |
| Approach: | They propose a framework for efficient serving of MoE-based large language models with tunable memory budgets. |
| Outcome: | Experiments show that SwapMoE can reduce memory consumption while maintaining reasonable accuracy. |
Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Sparse Mixture-of-Experts (SMoE) architectures require loading all expert parameters . previous work focused on expert pruning and merging but focused on neuron-level structure . |
| Approach: | They propose a task-agnostic framework for expert pruning and reconstruction . it prunes redundant experts using router statistics, then decomposes them into neuron-level expert segments . |
| Outcome: | The proposed framework reduces the number of experts and memory usage, making it easier to deploy. |
COMPEL: Compensated Mixture-of-Experts Pruning with Expert-Layer distribution (2026.findings-acl)
Copied to clipboard
| Challenge: | Mixture-of-Experts (MoE) architectures are effective for scaling Large Language Models (LLMs) however, existing pruning methods adopt uniform pruning across layers, which fails to capture layer-wise variations in expert importance and redundancy. |
| Approach: | They propose a Mixture-of-Experts pruning method that activates only a subset of experts during inference by estimating expert importance using Fisher information. |
| Outcome: | The proposed pruning method outperforms existing pruning methods while reducing inference latency and peak GPU memory usage. |